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1  Introduction 


Either  being  short  on  statisticians  or  employing  a  clever  marketing  scheme, 
Nabisco  has  recently  issued  the  “Chips  Ahoy!  1000  Chips  Challenge”  [1].  The 
cookie  company  is  asking  for  the  most  “creative”  way  to  confirm  that  there  are 
1000  chips  in  every  18  ounce  bag  of  their  chocolate  chip  cookies.  This  sounds 
like  a  simple  enough  challenge  but  several  questions  come  to  mind: 

1.  Where  do  we  get  the  bags  of  cookies? 

2.  How  many  bags  do  we  need? 

3.  How  do  we  count  the  chips  in  a  cookie? 

4.  What  method  of  data  analysis  is  most  appropriate? 

5.  Who  possibly  has  enough  free  time  to  partake  in  such  a  trivial  pursuit? 

Faculty  members  at  the  United  States  Air  Force  Academy  are  encouraged  to 
involve  cadets  in  interesting  and  sometimes  relevant  projects.  Likewise,  cadets 
are  always  eager  to  accept  a  challenge,  especially  if  chocolate  chip  cookies  are 
involved.  Thus,  we  have  the  perfect  mix  for  tackling  the  Chips  Ahoy!  challenge. 

This  paper  will  discuss  how  our  Introductory  Probability  and  Statistics 
course  addressed  the  above  questions  in  meeting  the  challenge.  However^  most 
of  the  attention  will  focus  on  the  fourth  question,  the  analysis,  since  as  statis¬ 
ticians,  this  is  our  bread  and  butter,  or  milk  and  cookies.  The  first  attempt 
to  analyze  the  data  involved  a  confidence  interval  for  the  mean.  We  quickly 
realized  that  this  standard  method  presented  in  most  introductory  courses  was 
inappropriate.  We  began  to  investigate  other  intervals  which  are  appropriate 
for  the  problem  but  are  not  typically  covered  in  an  introductory  course.  We 
will  share  some  insights  on  confidence,  prediction,  and  tolerance  intervals  and 
use  the  Chips  Ahoy!  challenge  to  tie  these  concepts  together. 

2  Methods 

An  18  ounce  bag  of  Chips  Ahoy!  cookies  contains  approximately  16  servings  at  3 
cookies  per  serving.  Nabisco  claims  that  each  bag  of  cookies  contains  over  1000 
chips.  A  chip  is  defined  as,  “any  distinct  piece  of  chocolate  that  is  baked  into 
or  on  top  of  the  cookie  dough  regardless  of  whether  or  not  it  is  100%  whole.”  [1] 

In  order  to  obtain  a  representative  sample  of  cookies,  cadets  had  friends  and 
relatives  from  across  the  country  mail  in  bags  of  cookies  to  the  Academy.  In 
all,  275  bags  were  received  from  46  different  states.  Once  all  the  cookies  were 

received  from  across  the  United  States,  it  was  time  to  count  the  chips.  Chips 
were  separated  from  the  cookie  by  dissolving  the  cookie  in  water.  Although  the 
preferred  method  of  cookie  extraction  is  to  dissolve  the  cookie  in  milk,  water 
was  used  to  minimize  costs. 
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From  the  275  bags  of  cookies  a  random  sample  of  42  bags  was  selected  for 
counting.  The  remaining  233  bags  were  used  to  maintain  the  energy  level  of  the 
cadets  doing  the  counting.  Data  for  the  number  of  chips  found  in  42  bags  of 
cookies  can  be  found  in  Table  1. 


Table  1:  Raw  Data  for  Number  of  Chips  per  Bag 
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3  Results 

Now  that  the  “cookie  is  crumbled”,  what  do  we  do?  In  this  section  we  will 
discuss  6  different  approaches  to  answering  this  question.  In  general  the  ap¬ 
proaches  involve  either  a  confidence  interval,  prediction  interval,  or  tolerance 
interval. 

3.1  Confidence  interval  for  the  mean  approach 

Being  in  an  introductory  statistics  course,  the  cadets  immediately  felt  that  a 
confidence  interval  would  be  the  most  appropriate  analysis,  A  lower  confidence 
bound  for  the  average  number  of  chips  in  a  bag  can  be  foimd  by 

—  4. 

y  ^1— a,n— 1  7=* 

y/n 

where  ti-a.n-iis  the  100(1— percentile  of  a  t  distribution  with  n  — 1  degrees 
of  freedom,  y  is  the  sample  mean,  Sy  is  the  sample  standard  deviation,  and  n 
is  the  sample  size.  For  our  data  set  the  95%  lower  confidence  bound  is  1231 
chips.  Therefore,  we  are  95%  confident  that  the  average  number  of  chips  per 
bag  is  at  least  1231.  By  confident  we  mean  that  95%  of  similarly  constructed 
intervals  from  many  independent  random  samples  would  actually  capture  the 
true  average  number  of  chips. 

There  are  several  draw  backs  with  this  method.  First,  it  is  an  estimate 
of  the  average  number  of  chips  per  bag.  Nabisco’s  claim  centers  aroxmd  the 
number  of  chips  in  an  individual  bag  not  on  the  average  number  of  chips.  That 
is,  Nabisco  is  interested  in  the  entire  distribution  of  chips  not  just  where  the 
distribution  is  centered.  Second,  the  interval  does  not  address  the  issue  of  1000 
chips.  Therefore,  even  though  the  confidence  interval  looks  promising,  it  really 
does  not  meet  the  challenge. 
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3.2  Confidence  interval  for  the  proportion  conforming 

After  scratching  our  heads  and  eating  a  few  cookies,  we  decided  to  look  at  a 
confidence  interval  for  the  proportion  of  bags  with  more  than  1000  chips.  From 
the  data  in  Table  1  we  can  see  that  100%  of  the  bags  have  over  1000  chips.  If 
we  define  the  random  variable  P  as  the  proportion  of  bags  with  more  than  1000 
chips,  we  can  find  a  lower  confidence  bound  for  tt,  the  true  population  propor¬ 
tion.  Most  introductory  statistics  courses  develop  the  confidence  bound  for  the 
proportion  based  on  the  normal  approximation.  However,  this  approximation 
is  poor  for  small  or  large  values  of  tt.  An  exact  lower  confidence  bound  can  be 
found  using 

1  —  /3(1  —  a,  n  —  x  -j- 1,  x) 

where  (3{\  —  a,  a,  6)  is  the  100(1  —  a)th  percentile  of  a  Beta  distribution  with 
parameters  a  and  b  [2].  Most  popular  spread  sheets  such  as  Microsoft  Excel 
have  this  function  built  in.  For  the  our  data  n  :=  42  and  x  =  42,  yielding 
a  95%  lower  confidence  bound  of  0.93.  Thus  we  are  95%  confident  that  the 
true  proportion  of  bags  with  more  than  1000  chips  is  at  least  93%.  Again,  by 
confident  we  mean  that  95%  of  similarly  constructed  intervals  will  captme  the 
true  population  proportion  of  bags  with  more  than  1000  chips. 

Their  are  two  potential  problems  with  this  approach.  First,  the  lower  boimd 
of  93%  seems  to  be  very  conservative.  Perhaps  by  making  some  distributional 
assumptions  a  tighter  bound  can  be  obtained.  Second,  the  defined  limit  (1000 
chips)  is  less  than  our  smallest  observed  value,  thus  the  lower  confidence  bound 
is  not  sensitive  to  this  limit.  In  other  words,  suppose  that  the  Nabisco  challenge 
had  been  for  more  than  500  chips  per  bag.  The  analysis  on  our  data  set  would 
allow  us  to  conclude  that  we  are  95%  confident  that  the  proportion  of  bags  with 
more  than  500  chips  is  at  least  93%.  Intuitively  one  would  think  that  we  should 
have  a  larger  proportion  conforming  as  the  limit  is  reduced. 


3.3  Prediction  interval 

Although  we  are  not  dealing  with  fortune  cookies,  another  approach  to  this 
problem  is  to  try  and  predict  the  number  of  chips  in  future  purchases  of  bags  of 
cookies.  One  way  to  do  this  is  with  a  prediction  interval.  The  prediction  interval 
is  based  on  the  difference  between  the  next  observation  and  the  mean.  Since 
these  variables  are  independent,  the  variance  of  this  difference  is  cr^^l  -f  1. 
Thus  the  appropriate  lower  prediction  bound  is 


y  ^l->a,n— l^y 


(1) 


This  interval  is  based  on  the  assumption  that  individual  observations  are  nor¬ 
mally  distributed  [3].  Figure  1  is  normal  Q-Q  plot  of  the  number  of  chips  per 
bag.  This  figure  along  with  a  Kolmogorov-Smirnov  goodness  of  fit  test  [4], 
reveals  there  is  no  reason  to  reject  the  claim  that  the  data  are  normally  dis¬ 
tributed.  Therefore,  we  are  95%  confident  that  the  next  bag  of  Chips  Ahoy! 


3 


Figure  1:  Normal  Q-Q  Plot  of  the  Number  of  Chips  in  a  Bag  of  Cookies. 


cookies  will  contain  more  than  1061  chips.  In  this  case  confident  means  95%  of 
similarly  constructed  prediction  intervals  from  independent  samples  will  contain 
the  future  value  from  another  independent  sample. 

A  problem  with  this  method  is  that  it  is  based  on  the  next  bag  of  cookies 
and  does  not  account  for  the  population  of  bags  of  cookies.  This  method  can 
be  extended  to  the  next  m  bags  of  cookies  using  a  Bonferonni  approach  where 
Equation  1  becomes  [5] 


Notice  that  the  width  of  the  interval  increases  with  the  number  of  predicted 
bags  m.  Another  problem  is  that,  again,  this  approach  doesn’t  focus  on  1000 
chips. 


y  ^l—afm,n—  lSy\  I + 


n 


3.4  Parametric  tolerance  interval 

After  consuming  several  bags  of  cookies,  our  tolerance  for  chocolate  chip  cookies 
was  gone.  But,  it  occurred  to  us  that  a  tolerance  interval  may  be  appropriate 
for  this  analysis.  A  tolerance  interv^  establishes  bounds  between  which  a  fixed 
proportion  of  individual  observations  will  occur.  If  we  knew  that  the  total 
number  of  chips  per  bag  was  normally  distributed  with  mean,  and  variance, 
then  95%  of  the  bags  of  cookies  will  contain  more  than  ji  —  1.645cr  chips. 
Since  we  do  not  know  ji  and  we  must  estimate  them  with  sample  statistics. 
The  use  of  the  sample  statistics  in  place  of  the  population  parameters  adds 
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uncertainty  in  the  proportion  of  the  population  covered  by  the  estimate  y  — 
The  tolerance  interval  accounts  for  this  uncertainty  by  assigning  a 
level  of  confidence  to  the  proportion  of  the  population  falling  in  the  interval. 
If  the  parent  population  is  normally  distributed,  the  tolerance  interval  has  the 
form 

y  -  ksy  (2) 

where  k  adjusts  for  the  use  of  sample  statistics  in  place  of  population  parameters. 
As  the  sample  size  (n)  increases,  k  approaches  the  standard  normal  percentile 
(for  a  5%  level  of  significance,  as  n  approaches  oo,  k  decreases  to  1.645).  Tables 
for  k  can  be  found  in  Eisenhart  [6].  For  our  data,  we  can  be  95%  confident  that 
at  least  95%  of  the  bags  of  Nabisco  Chips  Ahoy!  cookies  contain  more  than 
1012  chips.  It  is  important  to  note  that  the  claim  is  at  most  5%  of  all  bags 
of  cookies  will  contain  less  than  1012  chips.  By  confident  we  mean  that  95%  of 
similarly  constructed  intervals  from  independent  random  samples  would  cover 
at  least  95%  of  the  total  number  of  chips  per  bag. 

Notice  that  the  tolerance  statement  centers  on  1012  chips  and  not  1000. 
This  method  also  assumes  normality  for  the  distribution  of  the  number  of  chips 
in  a  bag  of  cookies.  While  this  assumption  is  plausible  for  this  data  set  (see 
figure  1),  one  may  not  always  be  willing  to  assume  normality.  In  which  case  a 
distribution-free  method  may  be  preferable. 


3.5  Distribution-free  tolerance  interval 

To  avoid  the  assumption  of  normality  in  the  parent  population,  a  distribution- 
free  tolerance  interval  based  on  order  statistics  can  be  formulated.  Based  on  the 
first  order  statistic,  the  smallest  value  in  our  data  set,  the  lower  tolerance  interval 
has  the  following  surprisingly  simple  relationship  (see  appendix  for  details): 

=  a.  (3) 

Here  p  is  the  proportion  of  the  population  greater  than  the  smallest  data  point 
(first  order  statistic),  a  is  the  level  of  significance,  and  n  is  the  sample  size. 
The  distribution-free  method  will  be  demonstrated  on  our  data  even  though 
the  normality  assumption  is  plausible.  This  is  done  for  demonstration  purposes 
only.  For  our  data  the  first  order  statistic  is  1087  chips;  using  a  .05  level  of 
significance  in  Equation  3  yields  a  value  of  .93  for  p.  Thus  we  are  95%  confident 
that  at  least  93%  of  all  bags  of  Chips  Ahoy!  cookies  contain  more  than  1087 
chips.  This  interval  is  exactly  the  same  as  we  would  have  obtained  by  the 
method  in  Section  3.2  if  we  had  defined  the  limit  as  1087  chips  instead  of  1000. 

Equation  3  also  provides  a  simple  formula  to  calculate  sample  size.  Solving 
for  n,  the  sample  size,  the  result  is: 


log^ 

logTrCP)  ' 


(4) 
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Figure  2:  Bootstrap  Distribution  of  the  Proportion  with  More  than  1000  Chips. 

3.6  Bootstrap  based  on  the  naive  approach 

A  relatively  simple  approach  that  first  semester  statistics  students  want  to  take 
is  to  calculate  P(Y  >  1000)  where  Y  is  the  number  of  chips  per  bag  of  cookies. 
This  probability  can  be  interpreted  as  the  proportion  of  all  bags  of  cookies  with 
more  than  1000  chips.  The  probability  calculation  is  based  on  the  assumption 
that  Y  is  normally  distributed  and  that  we  know  /z  and  cr^.  If  we  naively  use 
the  sample  statistics  as  estimates  of  the  parameters,  this  probability  calculation 
is  only  a  point  estimate  of  the  true  probability.  To  account  for  the  uncertainty 
associated  with  replacing  the  parameters  with  sample  statistics,  a  bootstrap 
method  [7]  was  used.  In  this  method,  we  sample  with  replacement  from  om 
original  data.  Based  on  this  sample,  we  calculate  the  sample  mean  and  sample 
variance  and  use  them  as  parameter  estimates  to  calculate  P{Y  >  1000)  where 
we  assume  Y  N{y,sl).  This  process  is  repeated  to  obtain  the  distribution 
of  the  proportion  of  bags  of  cookies  with  more  than  1000  chips.  A  bootstrap 
lower  confidence  bound  for  this  proportion  was  then  calculated.  Based  on  our 
data.  Figure  2  is  the  bootstrap  distribution  of  the  proportion  of  bags  with  more 
than  1000  chips.  The  5th  percentile  for  this  distribution  is  97%.  In  other  words, 
we  are  95%  confident  that  at  least  97%  of  all  bags  of  Chips  Ahoy!  cookies 
wiU  contain  more  than  1000  chips.  Notice  that  the  interpretation  is  similar  to 
Section  3.2;  however,  because  of  the  normality  assumption  the  confidence  of  this 
section  is  less  conservative;  it  is  narrower. 
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4  Discussion 


One  of  the  fascinating  and  fun  aspects  of  statistics  is  the  many  approaches  that 
can  be  used  to  solve  the  same  problem.  These  varied  approaches  often  have 
very  subtle  differences.  In  this  paper,  we  were  interested  in  finding  out  if  all 
18  ounce  bags  of  Chips  Ahoy!  cookies  contain  more  than  1000  chips.  We  first 
collected  bags  of  cookies  from  across  the  United  States  and  then  extracted  the 
chips  in  42  randomly  selected  bags.  For  the  analysis,  we  looked  at  confidence 
intervals,  prediction  intervals  for  a  single  bag,  tolerance  intervals  (parametric 
and  distribution- free),  and  a  bootstrap  confidence  bound  for  the  proportion 
of  bags  exceeding  the  specified  1000  chips  (for  an  excellent  discussion  of  the 
different  types  of  intervals  see  Hahn  and  Meeker  [5]).  To  reiterate,  the  results 
are  as  follows: 

1.  Using  confidence  intervals  for  the  mean,  we  can  be  95%  confident  that  the 
average  number  of  chips  per  bag  is  more  than  1231  chips. 

2.  Using  confidence  intervals  for  the  proportion  exceeding  a  limit,  we  can  be 
95%  confident  that  the  proportion  of  bags  with  more  than  1000  chips  is 
at  least  93%. 

3.  Using  a  prediction  interval,  we  can  be  95%  confident  that  the  next  single 
bag  of  Chips  Ahoy!  cookies  will  contain  more  than  1061  chips. 

4.  The  parametric  tolerance  interval  suggests  that  we  can  be  95%  confident 
that  at  least  95%  of  the  bags  contain  more  than  1012  chips. 

5.  The  nonparametric  tolerance  interval  suggests  that  we  can  be  95%  confi¬ 
dent  that  at  least  93%  of  the  bags  contain  more  than  1087  chips. 

6.  Using  the  bootstrap  approach  we  can  be  95%  confident  that  at  least  97% 
of  the  population  of  Chips  Ahoy!  cookie  bags  contain  more  than  1000 
chips. 

It  is  reassuring  to  see  that  all  the  methods  yield  consistent  answers.  However 
the  question  still  remains,  did  we  meet  the  Nabisco  challenge?  From  a  statistical 
point  of  view  we  need  to  acknowledge  that  we  can  never  be  100%  confident  that 
all  the  bags  contain  more  than  1000  chips.  As  an  example,  say  Nabisco  wants  to 
be  99.99%  confident  that  99.9999%  of  the  bags  of  Chips  Ahoy!  cookies  exceed 
the  smallest  observed  count.  Using  Equation  4,  they  would  need  to  count  the 
chips  in  9,210,336  bags  of  cookies.  If  all  of  the  9.2  milhon  bags  had  over  1000 
chips,  they  still  would  not  be  100%  confident  that  all  bags  have  over  1000  chips. 
However,  from  a  practical  point  of  view,  the  bootstrap  approach  states  that  we 
are  95%  confident  that  at  least  97%  of  the  population  of  Chips  Ahoy!  cookie 
bags  contain  more  than  1000  chips.  So,  yes  we  did  meet,  beat,  defeat,  and  eat 
the  Nabisco  Chips  Ahoy!  challenge! 
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Appendix 

The  distribution  of  the  order  statistic  [8]  is: 


oiVv)  = 


(r 


n! 

l)!(n  —  r)! 


(5) 


where  n  is  the  sample  size  and  f{x)  is  the  probability  density  function  of  the 
parent  random  variable  X.  Define  a  new  random  variable  p  as  the  proportion 
of  the  population  that  is  greater  than  the  order  statistic,  yr'. 


p<x> 

p  —  f{x)dx. 
Jvt 


(6) 


Transforming  Equation  5  using  Equation  6,  yields  the  probability  density  func¬ 
tion  of  p  as 

=  (7) 

Equation  7  is  a  Beta  distribution  with  parameters  n  -  r  -|- 1  and  r.  Notice  that 
Equation  7  does  not  depend  on  the  distribution  of  X!  For  the  case  of  the  first 
order  statistic  (r  =  1),  Equation  7  becomes 

h{p)  =  0  <  p  <  1.  (8) 


8 


To  find  the  proportion  p  of  the  population  that  exceeds  the  order  statistic 
with  (1  —  0:)%  confidence,  use  Equation  8  in 


which  reduces  to 


p”  =  a. 
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